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1. Introduction 


The worldwide rapid spread and severity of the infectious disease caused by Coronavirus 
forced the WHO to declare a global state of pandemic emergency during March 2020, by leading 
the governments around the world to adopt policies that created the widest rift of education 
systems in human history. As 85% of worldwide countries, also Italy has temporarily closed each 
educational institution, by causing the disruption of tertiary education for 16.89% of the Italian 
learner’s population. 

To ensure the “pedagogic continuity”, universities adopted the transitioning from traditional 
face-to-face to online learning (e.g., Tallent-Runnels et al., 2006; Sangrà et al., 2012; Todri et al., 
2021). In particular, the shift to fully remote teaching solutions as response to crisis is called by 
Hodges et al. (2020) as emergency remote teaching (ERT). This paradigm shift created changes 
about the perception of the learning process (Lederman, 2020) and supposed significant didactic 
efforts in terms of digitalisation and interactive pedagogical approaches. The ERT main goal is 
not to re-design a long-term educational ecosystem, but to supply a rapid and temporary solution 
to a crisis condition (Appolloni et al., 2021) adopting a learning framework different from online 
ones. 

This implicates venturing into uncharted territory with several logistical challenges and attitudinal 
modifications (Ribeiro, 2020), also in terms of teaching-learning assessment. 

Thus, the evaluation of ERT on the quality of higher education becomes a sensitive issue. This 
paper raises the evaluation of the effectiveness of teaching delivery during the transition from a 
traditional model to the ERT one. The focus is to detect how ERT is perceived and how it can 
connect to students’ performance and to the quality of education. The aspect of quality has been 
dealt out in terms of European Standards and Guidelines (ESG) adopted in 2005 by the Ministers 
of Higher Education of the countries participating in the Bologna Process (1999; Grano and Ricci, 
2009). To ensure the quality of the tertiary education system, European countries have established 
monitoring agencies. In Italy, this agency is ANVUR, which received official accreditation by the 
European association for quality assurance in higher education (ENQA) in 2019. However, during 
the Coronavirus health emergency, ANVUR did not provide guidance to universities on how to 
manage distance learning and its evaluations, relying on the autonomy of universities which 
continued to adopt the traditional evaluation systems. In a higher education landscape dominated 
by quality assurance view for evaluating teaching quality and student satisfaction (Fabbris, 2007), 
the teaching-learning ERT solutions adopted could not be fit since it is affected by several new 
factors and the principles recalled by the Bologna Process may be not appropriate. 

Thus, the paper focuses on an alternative simple tool for evaluating the quality of teaching- 
learning in ERT cases. Our research question has an explorative nature: we are interested in 
detecting empirical evidence about the learning assessment and engagement in higher education 
with focus on students’ engagement and their success performance during ERT. 

These dimensions have been represented in the ERT map inspired by perceptual maps of the 
consumer’ theory (Whitlark & Smith, 2001; Gower et al., 2010). In our model, the ERT map has 
been realised by a data integration perspective which considers the university administrative and 
textual data in a multivariate scenario of methodologies. Textual information is represented by the 
student voice, since this provides essential information for Quality Assurance systems and for 
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monitoring and managing information for universities processes. It represents one of the central 
issues in the most recent version of the European Guidelines and Standards for Quality Assurance 
in the European Higher Education Area adopted at the Yerevan meeting in 2015 and in AVA 
2017 (where AVA stands for Self-assessment, Evaluation, Accreditation, ANVUR, 2021). 

In the following, section 2 introduces the theoretical framework; section 3 describes our 
model and data; section 4 shows the main findings; section 5 presents the future directions. 


2. Theoretical framework 


During the last two years, the ERT condition caused by the SARS-CoV-2 pandemic allowed to 
enrich the literature by several contributions aimed to propose methods and techniques to evaluate 
the aspects of online teaching-learning in higher education. 

Among approaches to evaluate engagement and performance, Bawa (2020) examined the 

effects of the pandemic related ERT on leamers’ grades, using an experimental design to 
investigate the shift of online learning. The analysis has been realised by comparing the same 
course content and assessment methods for an experimental group formed of students enrolled 
during 2019-2020 and a control group with students who attended college before the health crisis. 
The results showed better outcomes in the experimental group than in the control group, above all 
for highest range performance. Dost et al. (2020) investigated attendance and perceptions of 
medical students across 40 UK schools during May 2020. By means of a cross-sectional study 
conducted on a national level via an online survey on a 20-items questionnaire measured on Likert 
scales, the study examined the experiences of online teaching, perceived benefits and barriers and 
the reached outcomes. From the findings, it emerges that online teaching platforms allow students 
to digest information in their own time and at the same time to discuss them with peers and, 
showed to be effective in terms of achieving learning outcomes. Huang et al. (2020) analysed the 
students’ engagement by adopting a mixed-methods design: from a quantitative descriptive 
approach to a qualitative visual method. 
At the end of course, all students were given a format inspired by the Motivated Strategies of 
Learning Questionnaire evaluated on Likert scales about four components: task value, 
metacognitive self- regulation, effort regulation, and peer learning. In addition, the demographic 
information and the answers to open- ended questions were coded and grouped into themes by 
qualitative approach. The results demonstrated the engagement does not depend on learning 
experience but on extrinsic goal orientation. 

Therefore, in a landscape of emerging difficulties to evaluate the student’s performance and 
engagement in ERT contexts as the Coronavirus one, our work proposes a strategy of quantitative 
analysis aimed to show empirical evidence about the learning assessment and engagement in 
higher education. 


3. Model and data 


To assess the quality and ERT success our proposal is based on study of two dimensions: the 
students’ engagement (SE) and success performance (SP) that represent the proxy variables used 
for constructing our model of analysis. SE has a textual nature comes from students’ voice whilst 
SP uses the career data of students. We obtained our model (Fig.1) by integrating these sources of 
information: textual ones linked to the analysis of the strengths of the engagement and 
administrative ones related to the results of students’ performance. We operated a 
multidimensional analysis to study the data from these different sources. 
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Fig.1: Model flowchart 


The measurement of the SE was focused on answers referring to the strengths of the 
engagement which have been analysed by means of the textual approach. We applied bags of 
words scheme to transform unstructured texts data in a structured data matrix to analyse. We 
operated the following pre-treatment operations: 1. Texts normalization; 2. Lemmatization; 3. 
Stopwords deleting. We built a Document-Term Matrix (DTM) without low frequency words 
(frequency cut at 5 minimum term frequency and 2 at document frequency of the feature) and 
empty documents. The DTM has texts for each row (755) and words in columns (790). 

The SP measurement is connected to the quantification of the students’ results and for 
reaching this goal the administrative data were used. To construct the Success Performance 
Indicator (SPI), we considered the average of marks (M) and ECTS (European Credit Transfer 
and Accumulation System) or credits (C) achieved before and during the pandemic. More 
precisely, for measuring each variation of marks and ECTS we used (1): 


n, — MaC 
AMC, = Ma = oCh (1) 


where Mio and Cio denote the average of marks and ECTS obtained until February 2020 for 
each i-th student, where i=/, ... N; Mi; and Ci; are the average of marks and ECTS achieved from 
February to November 2020. 
Therefore, the SPI; was computed by (2): 
AMC 
max(M,C) 100 (2) 


SPI, = 


It considers each variation of average of marks and ECTS in relation to the maximum average 
of marks and ECTS (maximum of mark and ECTS number, 30x180=5400) reachable by 100. 

To simplify the next steps, we recoded the SPI considering the quartile (Q) of SPI for each i-th 
student: 


1 — low if SPI; S Qujsp1,) 
SPI, = 42 — medium if Qyisp,) < SPI S Qispi 
3 —high if SPI; > Quispi) 


The SPI can be interpreted as: low performance when SPI;is lower or equal to the first quartile 
(SPI: < Q1), medium performance if SPI; is between the first and the second quartile (Q; < SPI; < 
Q2); high performance in the case of SPI; is greater to the third quartile (SPI;> Q3). 

Using the SPI; recoded in terms of quartile and the lexical dimension of SE, we created a 
contingency table to cross the proxy variables of our model. We inserted into this matrix the 
performance information at the student level in the rows and lexical keyword extracted from 


63 


DTM in the columns. This was possible because the DTM matrix (from SE study) and SP 
measurement matrix had the same rows. With a multidimensional data analysis approach, we 
operated a dimensional reduction through Correspondence Analysis (CA). A factorial plan or 
perceptual map, where we plotted our features, has been created. With this strategy, we can study 
the association between the two dimensions taken into account in our model. The intersection of 
these two proxy variables on the factorial plan shows on the horizontal axis SE while on the 
vertical axis SP. According to our model three principal theoretical areas can be imagined: 
- low SE and SP identify the quality of ERT at first level denoting a situation of 
attention to the low level of engagement and performance; 
- medium SE and SP denote the quality of ERT at second level denoting a situation of 
equilibrium between engagement and performance 
- finally, high SE and SP represent the quality of ERT at third level denoting a situation 
of high engagement and performance. 

The population is composed by the students enrolled in a three-year degree course at Iulm 
University of Milan (V=5000) during the academic year 2019-2020. The survey on ERT 
weaknesses and strengths had a response rate equal to 14% of the population. The investigated 
variables are related to the students’ career: year of the course; type of high school; gender; 
average of the marks and ECTS obtained until February 2020, before the Coronavirus, and from 
February to November 2020, during ERT. 

The female students are overrepresented. They respectively represent the 82.3% of the whole 
student population, the overall students enrolled in the first year are 44.3%. The Iulm University is 
composed by three Faculties: the 68% of the respondents studied at the Faculty of 
Communication and other respondents are equally split in other faculties. 


4. Results 
The recoded SPI barplot is symmetric and as shown in Fig.2 (a). 


2 J SPI high 
materiale foc to 
7 * spostamenti Pendolare 


maggiore registrazioni pars 

~ rivedere meno: cnn 
ell prendere |€ZiONiprobiemi 
seguire casa nessuno 


SPI low -ai = studiare teams 


m capito i momento 
Š concetto meg | 10 au la = 
SPllow SPI SPI high gestione ONIINE didattica 
medium ag lon sloppare 
»tomare 
G eee a ot aa f 
SPI 
medium 
(a) (b) 


For the SE, 755 texts were parsed and tokenized from the corpus. The results are a set of 
strings containing words used in documents. Subsequently, we reduced language variability to 
avoid possible sources of noise and to improve the effectiveness of the next analytical steps. The 
step consisted in the normalization of words, spelling and brought back each inflected word in its 
canonical form. Finally, we pruned non-informative words and non-alphabetic characters from the 
texts. The vocabulary size consisted of 790 types. Subsequently, we reduced the dimensionality of 
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this matrix, we filtered sparse words (with a sparsity threshold of 2%). At the end of the process, 
each document was represented as a document-vector and the number of types was 190. The 
comparison wordcloud plot shows the most frequent terms for separate grouping level of SPI 
(Fig. 2 (b)). The wordcloud allows to capture some differences among the words related to the 
different level of SPI: terms like “riascoltare” “listen to the lesson again” characterized the low 
level of SPI, whilst terms with negative meaning in the SPI were in the high-level area. 

As we affirm in Par.3, a contingency table was created, and we obtained the factorial plan through 
the CA (Fig.3) where the first dimension explains 54.4% and represents the ERT success while 
the second one explains 45.6% and denotes the SE. We can see three sections characterized by 
SPI levels splitting the map in the three horizontal levels: low, middle and high. On the contrary, 
the SE can be read easily from right to left, where we find an individual student engagement and 
the collective students’ engagement respectively. At first glance, we discover that the individual 
engagement is at the high level of SPI. This puts the semantic dimension related to the individual 
experience close to the high and low performance factors, as far as an exploratory analysis is 
concerned. We want to highlight that the two polarities (high and low) referring to the 
performance are both in the same half plane. Obviously, both refer to two different ways of 
experiencing distance for students. The difference consists, for the high SPI, in the facilitated 
access to the technologies made available and the possibility of optimizing the time available for 
the study. The low performance is close to the properly teaching dimension and relative to the 
contents of the courses. 
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Fig.3: Factorial plan 


5. Future directions 


The work proposes the data integration approach to create the ERT map. The CA allows to 
explore this integration using the contingency table built on SPI and SE, proxies developed to 
detect the success of the performance and the level of student engagement respectively. Attention 
should be paid to the use of short texts, inspired by the dialogue on social media, which do not 
always allow- based on the Italian language- to extract the true underlying concept. For this 
reason, future developments are moving in two directions: creation of an Italian dictionary, 
specifically for evaluating ERT, and creation of indicators that can be used in an agile way for 
subsequent ERT evaluations. The indicators could also be useful for drop out screening and 
prevention by monitoring the level of collaborative engagement. 
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